Itential Automation Platform (IAP) provides event logs for system and application administrators, developers, and troubleshooters. These event logs provide an audit trail to understand system activity and diagnose problems. See Log file at Wikipedia.
This guide is for the following audiences:
- System and application administrators
- Troubleshooter
- Developer
Log Message Activity
The IAP supports the following log severity levels.
| Log Level | Description |
|---|---|
| error | The most severe level. Indicates a problem, such as a prematurely terminated program. |
| warn | Used for warning conditions. Warn events may or may not indicate a problem. |
| info | Used for informational messages. |
| debug | Used only when debugging problems or transactions. Provides detailed information on messages between IAP adapters and external systems. |
| trace | Trace events have a higher verbosity than debug. Trace events are uncommon. |
| spam | The least severe level. Spam events have the highest verbosity and are rare. |
System and Application Administrator
IAP log settings specify where logs are stored, log rotation behavior, and event severity.
To view log settings in IAP:
Go to Settings > Properties.
Click Logging to show log settings.
Click Save after entering your changes.
Logging Settings
{
"description": "Logging",
"log_max_files": 100,
"log_max_file_size": 10485760,
"log_level": "debug",
"log_directory": "/var/log/pronghorn",
"log_filename": "pronghorn.log",
"console_level": "warn"
}After making changes to the log settings, restart IAP.
Logging JSON Schema
Below is an example JSON schema describing loggerProps. See JSON Schema to learn more about JSON schema specification.
{
"$id": "https://www.itential.com/schemas/properties.json/loggerProps.json",
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"description": {
"$id": "/properties/description",
"type": "string",
"title": "Pronghorn Properties Tab Title",
"description": "Text shown for the logging tab in Pronghorn
Properties view accessed from Pronghorn menu
SETTINGS->PROPERTIES.",
"default": "Logging",
"examples": [
"Logging"
]
},
"log_max_files": {
"$id": "/properties/log_max_files",
"type": "integer",
"title": "Number of Saved Log Files",
"description": "The number of old log files automatically
preserved.",
"default": 0,
"minimum": 0,
"examples": [
100
]
},
"log_max_file_size": {
"$id": "/properties/log_max_file_size",
"type": "integer",
"title": "Maximum Log File Size",
"description": "When Pronghorn's log file reaches this size
in bytes, Pronghorn makes a backup and clears
the log file's contents.",
"default": 0,
"minimum": 0,
"examples": [
104857600
]
},
"log_timezone_offset": {
"$id": "/properties/log_timezone_offset",
"type": "number",
"title": "Timezone Offset",
"description": "A number specifying the offset from UTC.
Pronghorn event logs will write the messages'
timestamps using this property for the
timezone.",
"default": 0,
"examples": [
-5,
1,
2.5
]
},
"log_level": {
"$id": "/properties/log_level",
"enum": [ "error", "warn", "info", "debug", "trace", "spam" ],
"title": "Minimum Logged Severity Level",
"description": "Specifies which event messages will be logged. Messages
with a severity level equal to or higher than this
property are logged. Severity error is the most
severe; severity spam is the least sever. Severity
error events indicate a problem, such as a
prematurely terminated program. Severity warn
events are used for warning conditions; they may
or may not indicate a problem. Severity info events
are informational messages. Severity debug events
are for debug-level messages, contain information
normally of use only when debugging a program, and
provide detailed information on messages between
Pronghorn adapters and external systems. Severity
trace events offer higher verbosity than debug,
and they are uncommon. Severity spam events offer
the highest verbosity and are rare.",
"default": "",
"examples": [
"debug"
]
},
"log_directory": {
"$id": "/properties/log_directory",
"type": "string",
"title": "Log File Directory",
"description": "Directory, specified as an absolute path, where
log files are saved.",
"default": "",
"examples": [
"/var/log/pronghorn"
]
},
"log_filename": {
"$id": "/properties/log_filename",
"type": "string",
"title": "Log File",
"description": "Log file filename.",
"default": "",
"examples": [
"pronghorn.log"
]
},
"console_level": {
"$id": "/properties/console_level",
"enum": [ "error", "warn", "info", "debug", "trace", "spam" ],
"title": "Minimum Console Severity Level",
"description": "Specifies which event messages will be written
to the console. Messages with a severity level
equal to or higher than this property are
logged. Severity error is the most severe;
severity spam is the least sever. Severity
error events indicate a problem, such as a
prematurely terminated program. Severity warn
events are used for warning conditions; they
may or may not indicate a problem. Severity
info events are informational messages.
Severity debug events are for debug-level
messages, contain information normally of
use only when debugging a program, and provide
detailed information on messages between
Pronghorn adapters and external systems.
Severity trace events offer higher verbosity
than debug, and they are uncommon. Severity
spam events offer the highest verbosity and
are rare.",
"default": "",
"examples": [
"warn"
]
}
}
}Severity Thresholds
Production environments should generally have the log_level set to warn or info. Debug, trace, and spam log levels will likely generate a large amount of log data and additional server load. Only configure production servers in debug mode at the request of Itential Support.
For systemd operating systems, the system journal manages console logging. The system journal may also contain application life cycle error messages that cannot be saved to the IAP file logs.
Log File Retention
IAP provides native support for rotation of the pronghorn.log file. Administrators can define the total allocated storage for logs by configuring the maximum file size and maximum number of files. For example, if the maximum file size is 1 MB and the total number of files is 100, the total space consumed by pronghorn.log files will be 100 MB. Make sure the disk that holds the pronghorn logs contains enough space for IAP to exercise a full log rotation cycle.
Consider both IAP file logs and system logs when planning server storage requirements.
Temporal Severity Threshold Settings
Every IAP adapter and application inherits the system log and console severity thresholds when IAP starts. After IAP starts, every adapter and application can be optionally configured with local, temporal log and console severity thresholds. The temporal thresholds are reset to IAP global thresholds when IAP restarts.
- Go to Settings > System.
- Click on an Application or Adapter row to open the module’s Log Settings view.
- Set Temporal File Log Level.
- Set Console Log Level.
- Click Update.
Troubleshooting
If a problem prevents IAP from loading the global log object from the Logger class, log messages will not reach the IAP log file. Log messages will, however, reach the systemd journal or /var/log/messages on System V (“System Five”) hosts.
Monitor the system journal for errors and warnings using shell command journalctl -f. If many processes are sending message to the system journal, filter log messages to show only IAP logs using shell command journalctl -f -u pronghorn.service.
See Red Hat systemd Journal Documentation for more information.
Monitor IAP logs with a tail follow shell command, such as tail -F /var/log/pronghorn/pronghorn.log.
Reading the Logs
IAP logs are JSON objects that are better for automating log parsing operations compared to simple text.
Log Object Properties
| Property | Description |
|---|---|
@timestamp |
Automatically added. |
@fields.origin |
The message location: application that created the log message and its line number associated with the event. |
@fields.level |
The event message severity level. |
@message |
An array. Elements can be strings or objects. An array with one string element containing log message text that can be a stack trace, an error received from an external system, or a log message the IAP developer sent with the global log object. |
Troubleshooting a Layered Architecture
IAP applications exist in a layered architecture. IAP has multiple layers, and it is one of many autonomous systems that collectively create network automation. The best way to troubleshooting multiple layers is to establish control and eliminate variables.
For example, suppose the application method called from a job task is failing. The task coordinates with an external network controller to affect device changes. Begin by verifying the network controller is able to execute the exact requested action. If the controller has a CLI, that is the best place to verify an action. If the CLI test passes, try calling the action through the controller’s API. If the controller API test passes, use a web API client, such as Google Postman, to test the IAP application method without the IAP workflow engine. Finally, use the Job Manager IAP application to inspect run-time data: passed arguments, data or errors returned.
Developer
Adapter and application developers can log events using the IAP global log instance of the Logger class. The log methods specify the event severity. Each takes an argument, a string to be saved in the logs.
Best Practices
- Inside functions, log non-sensitive passed arguments at debug severity.
- Inside functions, log non-sensitive returned data at debug severity.
Examples
| Event | Method | Message |
|---|---|---|
| Error Severity | log.error() |
This message will be recorded as an error. It will be sent for all severity levels. |
| Warning Severity | log.warn() |
This message will be recorded as a warning. It will be sent for severity levels warn, info, debug, trace, and spam. |
| Informational Severity | log.info() |
This message will be recorded as informational. It will be sent for severity levels info, debug, trace, and spam. |
| Debug Severity | log.debug() |
This message will be recorded as debug. It will be sent for severity levels debug, trace, and spam. |
| Trace Severity | log.trace() |
This message will be recorded as a trace. It will be sent for severity levels trace and spam. |
| Spam Severity | log.spam() |
This message will be recorded as spam. It will only be sent for severity level spam. |